{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "rV7GJkLUqpcW"
      },
      "source": [
        "# Monitoring Hugging Face LLMs with LangKit\n",
        "\n",
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/langkit/blob/main/langkit/examples/huggingface_langkit_whylabs.ipynb)\n",
        "\n",
        "In this example, we'll show how to generate out-of-the-box text metrics for Hugging Face LLMs using LangKit and monitor them in the WhyLabs Observability Platform.\n",
        "\n",
        "LangKit can extract relevant signals from unstructured text data, such as:\n",
        "\n",
        "- [Text Quality](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/quality.md)\n",
        "- [Text Relevance](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/relevance.md)\n",
        "- [Security and Privacy](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/security.md)\n",
        "- [Sentiment and Toxicity](https://github.com/whylabs/langkit/blob/main/langkit/docs/features/sentiment.md)\n",
        "\n",
        "We'll use the GPT2 model for this example since it's lightweight and easy to run without a GPU, but any of the larger Hugging Face models can be used."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "UjKQIAHDqv3q"
      },
      "source": [
        "### Install Hugging Face Transformers & LangKit"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5rfQw6fBnGnC"
      },
      "outputs": [],
      "source": [
        "%pip install transformers\n",
        "%pip install 'langkit[all]'"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "xxBB-BDsqw-P"
      },
      "source": [
        "### Import and initialize the Hugging Face GPT2 model + tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wedbxmUaqzAc"
      },
      "outputs": [],
      "source": [
        "from transformers import GPT2LMHeadModel, GPT2Tokenizer\n",
        "\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
        "model = GPT2LMHeadModel.from_pretrained(\"gpt2\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "K8A5TvEGrOsC"
      },
      "source": [
        "### Create GPT model function\n",
        "This will take in a prompt and return a dictionary containing the prompt and model response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "XSrG3wNdw4SR"
      },
      "outputs": [],
      "source": [
        "def gpt_model(prompt):\n",
        "\n",
        "  # Encode the prompt\n",
        "  input_ids = tokenizer.encode(prompt, return_tensors='pt')\n",
        "\n",
        "  # Generate a response\n",
        "  output = model.generate(input_ids, max_length=100, temperature=0.8,\n",
        "                          do_sample=True, pad_token_id=tokenizer.eos_token_id)\n",
        "\n",
        "  # Decode the output\n",
        "  response = tokenizer.decode(output[0], skip_special_tokens=True)\n",
        "\n",
        "  # Combine the prompt and the output into a dictionary\n",
        "  prompt_and_response = {\n",
        "      \"prompt\": prompt,\n",
        "      \"response\": response\n",
        "  }\n",
        "\n",
        "  return prompt_and_response"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CROG5UHfzHMl"
      },
      "outputs": [],
      "source": [
        "prompt_and_response = gpt_model(\"Tell me a story about a cute dog\")\n",
        "print(prompt_and_response)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "l1lP4Ybgy2go"
      },
      "source": [
        "### Create & Inspect Language Metrics with LangKit\n",
        "\n",
        "LangKit provides a toolkit of metrics for LLM applications, lets initialize them and create a profile of the data that can be viewed in WhyLabs for quick analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hHFiivQIyyVv"
      },
      "outputs": [],
      "source": [
        "from langkit import llm_metrics # alternatively use 'light_metrics'\n",
        "import whylogs as why\n",
        "\n",
        "why.init(session_type='whylabs_anonymous')\n",
        "# Note: llm_metrics.init() downloads models so this is slow first time.\n",
        "schema = llm_metrics.init()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "04LMKv8gV4FW",
        "outputId": "753cf4fe-558b-4890-d7b6-b3b6762c28c8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Aggregated 1 rows into profile 'HF prompt & response'\n",
            "\n",
            "Visualize and explore this profile with one-click\n",
            "🔍 https://hub.whylabsapp.com/resources/model-1/profiles?sessionToken=session-8gcsnbVy&profile=ref-AVbB1yOaSblsa89U"
          ]
        }
      ],
      "source": [
        "# Let's look at our prompt_and_response created above\n",
        "profile = why.log(prompt_and_response, name=\"HF prompt & response\", schema=schema)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "MAnitqOGzZq3"
      },
      "source": [
        "Click the link generated above to view the language metrics in Whylabs.\n",
        "\n",
        "We can also see all our values by viewing our LangKit profile in a pandas data frame.\n",
        "\n",
        "You can use this data in real time to make a decision about prompts and responses, such as setting guardrails on your model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 268
        },
        "id": "VToO4A4DyyYU",
        "outputId": "281ad373-d900-4894-d62d-5e0dbc5a1c15"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-3950ea9c-cc22-4d1a-a352-5c7a34550331\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cardinality/est</th>\n",
              "      <th>cardinality/lower_1</th>\n",
              "      <th>cardinality/upper_1</th>\n",
              "      <th>counts/inf</th>\n",
              "      <th>counts/n</th>\n",
              "      <th>counts/nan</th>\n",
              "      <th>counts/null</th>\n",
              "      <th>distribution/max</th>\n",
              "      <th>distribution/mean</th>\n",
              "      <th>distribution/median</th>\n",
              "      <th>...</th>\n",
              "      <th>udf/toxicity:distribution/q_95</th>\n",
              "      <th>udf/toxicity:distribution/q_99</th>\n",
              "      <th>udf/toxicity:distribution/stddev</th>\n",
              "      <th>udf/toxicity:frequent_items/frequent_strings</th>\n",
              "      <th>udf/toxicity:types/boolean</th>\n",
              "      <th>udf/toxicity:types/fractional</th>\n",
              "      <th>udf/toxicity:types/integral</th>\n",
              "      <th>udf/toxicity:types/object</th>\n",
              "      <th>udf/toxicity:types/string</th>\n",
              "      <th>udf/toxicity:types/tensor</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>column</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>prompt</th>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.00005</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>0.005245</td>\n",
              "      <td>0.005245</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[FrequentItem(value='0.005245', est=1, upper=1...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>response</th>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.00005</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>0.004731</td>\n",
              "      <td>0.004731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[FrequentItem(value='0.004731', est=1, upper=1...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>response.relevance_to_prompt</th>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.00005</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.627636</td>\n",
              "      <td>0.627636</td>\n",
              "      <td>0.627636</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3 rows × 476 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3950ea9c-cc22-4d1a-a352-5c7a34550331')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-75b66d67-ea4a-43bd-a24e-c7976a6cfa08\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-75b66d67-ea4a-43bd-a24e-c7976a6cfa08')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-75b66d67-ea4a-43bd-a24e-c7976a6cfa08 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3950ea9c-cc22-4d1a-a352-5c7a34550331 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3950ea9c-cc22-4d1a-a352-5c7a34550331');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                              cardinality/est  cardinality/lower_1  \\\n",
              "column                                                               \n",
              "prompt                                    1.0                  1.0   \n",
              "response                                  1.0                  1.0   \n",
              "response.relevance_to_prompt              1.0                  1.0   \n",
              "\n",
              "                              cardinality/upper_1  counts/inf  counts/n  \\\n",
              "column                                                                    \n",
              "prompt                                    1.00005           0         1   \n",
              "response                                  1.00005           0         1   \n",
              "response.relevance_to_prompt              1.00005           0         1   \n",
              "\n",
              "                              counts/nan  counts/null  distribution/max  \\\n",
              "column                                                                    \n",
              "prompt                                 0            0               NaN   \n",
              "response                               0            0               NaN   \n",
              "response.relevance_to_prompt           0            0          0.627636   \n",
              "\n",
              "                              distribution/mean  distribution/median  ...  \\\n",
              "column                                                                ...   \n",
              "prompt                                 0.000000                  NaN  ...   \n",
              "response                               0.000000                  NaN  ...   \n",
              "response.relevance_to_prompt           0.627636             0.627636  ...   \n",
              "\n",
              "                              udf/toxicity:distribution/q_95  \\\n",
              "column                                                         \n",
              "prompt                                              0.005245   \n",
              "response                                            0.004731   \n",
              "response.relevance_to_prompt                             NaN   \n",
              "\n",
              "                              udf/toxicity:distribution/q_99  \\\n",
              "column                                                         \n",
              "prompt                                              0.005245   \n",
              "response                                            0.004731   \n",
              "response.relevance_to_prompt                             NaN   \n",
              "\n",
              "                              udf/toxicity:distribution/stddev  \\\n",
              "column                                                           \n",
              "prompt                                                     0.0   \n",
              "response                                                   0.0   \n",
              "response.relevance_to_prompt                               NaN   \n",
              "\n",
              "                                   udf/toxicity:frequent_items/frequent_strings  \\\n",
              "column                                                                            \n",
              "prompt                        [FrequentItem(value='0.005245', est=1, upper=1...   \n",
              "response                      [FrequentItem(value='0.004731', est=1, upper=1...   \n",
              "response.relevance_to_prompt                                                NaN   \n",
              "\n",
              "                              udf/toxicity:types/boolean  \\\n",
              "column                                                     \n",
              "prompt                                               0.0   \n",
              "response                                             0.0   \n",
              "response.relevance_to_prompt                         NaN   \n",
              "\n",
              "                              udf/toxicity:types/fractional  \\\n",
              "column                                                        \n",
              "prompt                                                  1.0   \n",
              "response                                                1.0   \n",
              "response.relevance_to_prompt                            NaN   \n",
              "\n",
              "                              udf/toxicity:types/integral  \\\n",
              "column                                                      \n",
              "prompt                                                0.0   \n",
              "response                                              0.0   \n",
              "response.relevance_to_prompt                          NaN   \n",
              "\n",
              "                              udf/toxicity:types/object  \\\n",
              "column                                                    \n",
              "prompt                                              0.0   \n",
              "response                                            0.0   \n",
              "response.relevance_to_prompt                        NaN   \n",
              "\n",
              "                              udf/toxicity:types/string  \\\n",
              "column                                                    \n",
              "prompt                                              0.0   \n",
              "response                                            0.0   \n",
              "response.relevance_to_prompt                        NaN   \n",
              "\n",
              "                              udf/toxicity:types/tensor  \n",
              "column                                                   \n",
              "prompt                                              0.0  \n",
              "response                                            0.0  \n",
              "response.relevance_to_prompt                        NaN  \n",
              "\n",
              "[3 rows x 476 columns]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "profview = profile.view()\n",
        "profview.to_pandas()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "R0Gy7-9z2tBE"
      },
      "source": [
        "## ML Monitoring for Hugging Face LLMs in WhyLabs\n",
        "\n",
        "\n",
        "To send LangKit profiles to WhyLabs we will need three pieces of information:\n",
        "\n",
        "- API token\n",
        "- Organization ID\n",
        "- Dataset ID (or model-id)\n",
        "\n",
        "Go to [https://whylabs.ai/free](https://whylabs.ai/free) and grab a free account. You can follow along with the quick start examples or skip them if you'd like to follow this example immediately.\n",
        "\n",
        "1. Create a new project and note its ID (if it's a model project, it will look like `model-xxxx`)\n",
        "2. Create an API token from the \"Access Tokens\" tab\n",
        "3. Copy your org ID from the same \"Access Tokens\" tab\n",
        "\n",
        "Replace the placeholder string values with your own OpenAI and WhyLabs API Keys below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Udn5aaFpomlN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "# set authentication & project keys\n",
        "os.environ[\"WHYLABS_DEFAULT_ORG_ID\"] = 'ORGID'\n",
        "os.environ[\"WHYLABS_API_KEY\"] = 'APIKEY'\n",
        "os.environ[\"WHYLABS_DEFAULT_DATASET_ID\"] = 'MODELID'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jsRrkZGT2J-p"
      },
      "outputs": [],
      "source": [
        "from whylogs.api.writer.whylabs import WhyLabsWriter\n",
        "from langkit import llm_metrics # alternatively use 'light_metrics'\n",
        "import whylogs as why\n",
        "\n",
        "# Note: llm_metrics.init() downloads models so this is slow first time.\n",
        "schema = llm_metrics.init()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aKP7jDcu2KBP"
      },
      "outputs": [],
      "source": [
        "# Single Profile\n",
        "telemetry_agent = WhyLabsWriter()\n",
        "profile = why.log(prompt_and_response, schema=schema)\n",
        "telemetry_agent.write(profile.view())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "yLxwVo503YG3"
      },
      "source": [
        "This will write a single profile to WhyLabs.\n",
        "\n",
        "As more profiles are written on different dates, you'll get a time series pattern you can analyze & set monitors like in the [Demo org](https://bit.ly/3NOq0Od).\n",
        "\n",
        "You can also backfill batches of data by overwriting the date and time as seen in [this example](https://github.com/whylabs/langkit/blob/main/langkit/examples/Batch_to_Whylabs.ipynb).\n",
        "\n",
        "![](https://github.com/whylabs/langkit/blob/main/static/img/sentiment-monitor.png?raw=1)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PDDSjr54BW-C"
      },
      "source": [
        "## Optional: Use a Rolling Logger\n",
        "A rolling logger can be used instead of the method above to write profiles at pre-defined intervals."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w4jz_x2GAXnP"
      },
      "outputs": [],
      "source": [
        "telemetry_agent = why.logger(mode=\"rolling\", interval=5, when=\"M\",schema=schema, base_name=\"huggingface\")\n",
        "telemetry_agent.append_writer(\"whylabs\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c2eoEc-zAXps",
        "outputId": "93362693-3502-4031-84a8-2119c45fe39d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<whylogs.api.logger.result_set.ProfileResultSet at 0x7f6f7c5b3af0>"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Log data + model outputs to WhyLabs.ai\n",
        "telemetry_agent.log(prompt_and_response)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PStG7ZtAAp7l"
      },
      "outputs": [],
      "source": [
        "# Close the whylogs rolling logger when the service is shut down\n",
        "telemetry_agent.close()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "24EqoHdgrBhQ"
      },
      "source": [
        "# More Resources\n",
        "\n",
        "Learn more about monitoring LLMs in production with LangKit\n",
        "\n",
        "- [Intro to LangKit Example](https://github.com/whylabs/langkit/blob/main/langkit/examples/Intro_to_Langkit.ipynb)\n",
        "- [LangKit LangChain Integration](https://github.com/whylabs/langkit/blob/main/langkit/examples/Langchain_OpenAI_LLM_Monitoring_with_WhyLabs.ipynb)\n",
        "- [LangKit GitHub](https://github.com/whylabs/langkit)\n",
        "- [whylogs GitHub - data logging & AI telemetry](https://github.com/whylabs/whylogs)\n",
        "- [WhyLabs - Safeguard your Large Language Models](https://whylabs.ai/safeguard-large-language-models)\n",
        "- [Hugging Face GPT2 Model](https://huggingface.co/gpt2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D5W1hUjQaKsz"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
